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EXECUTIVE SUMMARY 


A usability study was conducted by the Research and Innovation Network and Assessment 
Solutions Design to assess the usability of a drawing response interaction type. The Partnership 
for Assessment of Readiness for College and Careers (PARCC) commissioned a multi-part study 
to determine the viability of using the drawing response interaction on the PARCC Mathematics 
Assessment. In the first part of the study, participants in grades 3, 9, 10, and 11 (during the 
2014-2015 school year) participated. Participants completed a series of math questions on the 
computer or iPad that included the drawing response interaction, while observation and video 
recording took place. 


The goal of this study was to test the usability of the drawing response interaction feature and 

determine if and how it assists participants in problem solving. The team also wanted to gather 
feedback from the participants on how they would change the drawing response interaction to 
best help them solve problems. 


Positive findings from the study include the following: 


e Participants were generally comfortable using the drawing response interaction for 
basic tasks 

e Participants generally liked the drawing response interaction and thought they would 
use it when solving problems 


Issues revealed during this study include the following: 


Participants were generally confused by the eraser and had difficulty erasing their work 
Participants were very challenged by the math content and, at times, were unable to 
give feedback on the drawing response interaction because of this 

e Participants were frustrated by the drawing response interaction’s lack of precision, in 
both iPad and desktop/mouse solutions, particularly 3rd graders, whose fine motor skills 
are not well-developed 

e Participants were concerned with their drawing ability and how that might impact their 
test scores 


Recommendations for future studies: 


Adjust the eraser tool 
Include a library of shapes and other tools to help participants create more accurate 
work in the drawing response interaction 

e Adjust the rigor of math questions included 
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e Include clear instructions for how and where participants should show their work and 
answers 


INTRODUCTION 


The PARCC Tests assess participants’ current performance on rigorous state standards for math 
and Language Arts; the tests are designed to not only assess for correctness, but to also look for 
evidence that a participant understands and can apply their knowledge. The goal of the study 
was to gather feedback on a drawing response interaction, which may be integrated into the 
PARCC assessment. Pearson is working with PARCC to deliver a flexible drawing response 
interaction feature to enable participants to record and submit their handwritten work and 
answers for math problems. 


Purpose 

Pearson’s Assessment Solutions Design team wanted to gather information on the usability of 
the drawing response interaction on a computer and iPad and its usefulness to participants in 
problem solving. 


The team also recorded observations about physical interaction with the iPad and desktop 
computer, with a focus on the usability of the drawing response interaction. 


Test Objectives 
e Can participants use the drawing response interaction to help them solve math 
problems? 
e@ What features do participants want to add to the drawing response interaction? 


General Observation Objectives 
e Will participants be able to successfully complete the test questions? 
o Without moderator prompts? 
o With using the drawing response interaction? 
e Will participants encounter any difficulties with the drawing response interaction 
features? 
e Will participants encounter any general technological difficulties with the computer or 
iPad? 
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User Profile 
There were twelve total participants, including six third graders, four ninth graders, one tenth 


grader, and one eleventh grader. Note that grade levels were taken as of the 2014-2015 school 


year. All participants self-reported as comfortable (scores of 4 or 5 ona 1 to 5 scale) with math 
on the initial survey they filled out. 


Participant | Grade | Gender | Use of Test Tool Math Comfort Math Courses 
computer for (Computer or iPad) | Level (1=not Taken (for HS 
math? confident at all, participants) 

5=extremely 
confident) 

P1 9 Female No—only Computer 4 Algebra I, 
paper/pencil Geometry 

P2 3 Male No—but has Computer 4 
used art apps 

P3 3 Male No Computer Comfortable, did 

not ask rating 

P4 3 Female Yes—uses an Computer 4 
app for basic 
math at school 

PS 10 Male Yes—in his dual | Computer 4 Algebra |, 
enrollment Geometry, 
math course Algebra Il, 

Trigonometry 
P6 9 Female Yes—uses Computer 5 Algebra 1, 
WebWorks and Geometry 
has taken state 
tests online 
P7 3 Male No iPad 4 
P8 3 Female iPad 5 
P9 9 Male Yes—has used iPad 4 Algebra I, 
Galileo Geometry 
P10 11 Female iPad 4 Algebra I, 
Geometry, 
Algebra Il, Pre- 
Calculus 

Pil 3 Male iPad 4 

P12 9 Female Yes—uses iPad 5 Algebra I, 


Geometry, 
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online textbook Algebra II 


METHODS 


The usability test was completed within 60 minutes. The test was designed to analyze and 
measure the effectiveness of the drawing response interaction in helping participants solve 
math problems and gather feedback from the participants on what functionality they would 
add to the drawing response interaction. Each participant session was organized in the same 
way to facilitate consistency. Video and audio recordings and moderator/note-taker 
observations were used to collect data. 


Test Details 


Recruitment (pre-test) 


Participants were recruited through an initial recruitment email with a short explanation 
of the study, along with the dates and incentive information. Inside the email was a link 
to a survey that parents could fill out to determine math comfort. The survey included 
the following questions: 1) Rate your child's confidence level (1-5), with 1 being Not at 
all Confident and 5 being Extremely Confident, with activities related to mathematics 
(math games, math assignments, math tests, etc.), 2) (for high school participants only) 
From the list, please select the math courses that your child has completed: Algebra I, 
Geometry, Algebra II, Trigonometry, Calculus, Other. 


Introduction/Demographic Interview (2 - 5 minutes) 


The moderator explained that the purpose of the test was to review online tools 
embedded in a high-stakes assessment. Participants answered questions to get a sense 
of their past use of technology products in their math courses. These questions allowed 
us to gain an understanding of the participant’s previous use of technology with math 
while giving the participant a chance to get comfortable. Last, the moderator 
emphasized that we are not testing the participant’s ability to solve the math problems 
but rather their perceptions of the online tools. 


Prototype Testing / Follow Up Interview (25 - 45 minutes) 


The moderator explained that the participant should imagine that he/she is about to 
take a test online and asked them to pretend this was a high stakes test similar to one 
they would take at the end of the year. The moderator instructed the participant to 
“think aloud” as he/she completed the tasks to let us know what he/she was thinking. 
The test was delivered via a desktop computer for three of the third graders and three 
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of the high schoolers and via an iPad for the remaining three third graders and three 
high schoolers. 


Farewell (1 - 2 minutes) 
When the task list was finished, participants were prompted to provide any other 
relevant feedback. Participants received an honorarium at the end of the session. 


After test data had been collected, notes and video data were reviewed and coded to 
determine themes. 


Test Environment 

Usability testing was conducted in the Idea Innovation Center in the Pearson Chandler office. 
Participants sat at a table with a moderator. In person observers/note-takers sat behind the 
one-way mirror, and one remote observer/note-taker joined by phone and WebEx. For each 
session, the audio and screen movements were recorded using Ovo. 


Test Moderator/Observer Role 

Before the session, two of the observers/note-takers retrieved participants from the Chandler 
office lobby. They answered any questions the participant/guardian or parent had and then 
walked the participant to the Idea Innovation Center. 


The moderator met the participant in the Idea Innovation Center and introduced and 
administered the usability test, referencing a test script as needed. The moderator introduced 
the participants to the assessment and informed them of the new tool. The moderator 
explained that we were more interested in the participant’s perception of the tools than of 
their ability to answer the math questions correctly. The moderator explained the “think 
aloud” procedure for the session and answered any preliminary questions. The moderator 
administered the test and asked the participants to rate the tasks completed during the session 
on a scale of 1 to 7, 1 being very difficult and 7 being very easy. At the end of the session, the 
moderator presented the participant with a $50 Visa gift card. After the session, one of the 
observers/note-takers walked the participant back to the Chandler office lobby. 


During each interview, the observers operated the lab cameras and recording software. The 
observers took notes on the feedback from the sessions. After each interview, the moderator 
organized and posted video footage for review. 
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Test Format/Prototype Overview 
The test was divided into three phases. 


1, 


In the first phase, all participants interacted with three tasks which required them to 
draw basic shapes (a line, a circle, and a picture to illustrate one half). High school 
participants interacted with two additional tasks--graphing a line and drawing a 
geometric figure. These tasks were written specifically for this usability study and were 
designed to test whether or not participants could interact with the drawing response 
interaction on a basic level. 

In the second phase, participants interacted with a series of tasks that required them to 
solve (or attempt to solve) a question using the drawing response interaction. These 
questions were drawn from: a) PARCC assessment questions which participants 
including a drawing on the paper assessment and were modified for security reasons for 
this usability test, b) PARCC practice and sample test questions. For the third grade 
participants, tasks were drawn from grade 3 items, and for the high school participants, 
tasks were drawn from Algebra | and Geometry items. 

In the third phase, participants interacted with a series of tasks that were identified as 
candidates for inclusion of the drawing response interaction. For these tasks, 
participants were not asked to solve, but were asked if they would use the drawing 
response interaction, and if so, how they would use it and what features they would 
want added. 


Evaluation Measures 


For each usability study objective, the following measures will be used: 


Observation of users’ interactions with the drawing response interaction 

Verbal feedback during follow-up questions 

Participant ratings (on a scale of 1 to 7, 1 being very difficult, 7 being very easy) for each 
task 
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RESULTS 


Twelve participants participated in PARCC’s drawing response interaction usability study. Video 
and audio recordings are available for all twelve interviews (see Appendix 2). One participant 
became frustrated during the interview and decided to stop early; we included the responses 
and feedback she was able to provide. 


Results are broken down by task and grade level (third or high school). 


Previous Technology Usage with Mathematics 


Prior to beginning the usability test, participants were probed regarding their use of technology 
with math. 


e@ 30of 12 participants reported using some sort of drawing/art app for fun 
e 5 of 12 participants reported using some sort of technology in their math courses 
© P4 uses an app for basic math (adding, subtracting, multiplying, dividing) at 
school 
PS uses the computer for his dual enrollment math course 
P6 uses WebWork for her math homework and has taken state math tests on the 
computer, using tools such as rulers, calculators, and highlighters 
P9 uses Galileo in his math course 
P12 uses an online textbook for his math course 


Task-Level Analysis for Third Graders & High Schoolers 


General Perceptions of the Drawing Response Interaction 


In general, participants liked the drawing response interaction and could use it for simple 
tasks 


12 of 12 participants were able to draw a line using the drawing response interaction 
12 of 12 participants were able to draw a circle using the drawing response interaction 
P5: “I really like the drawing response interaction--it’s simple. Being able to draw it 


out helps a lot with problem solving.” 

P5: “Especially for me, math is easier using pictures so you can see what you are 
doing.” 

P9: “A drawing box is good for any [problem] so you can write down little notes.” 


In general, participants recognized the drawing response interaction features 
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@ 9of 12 participants verbally pointed out the color icons 
e 5of 12 participants verbally pointed out the eraser icon 
In general, participants wanted the drawing response interaction to be more accurate 

e P9: “I want [drawing response interaction] to be more precise.” 

e P12: “Drawing it out is a nice concept, but it’s a lot easier if the numbers are exact-- 
lines are exactly the same, etc.” 
P11: “I would like to write [my response] smaller and neater.” 
P12: “Honestly, it would be hard to draw a perfect circle.” 
P6: “It’s important for me to be able to draw a perfect circle.” 
P1: “Il don’t want to be judged off of my artistic skills.” 


3 of 12 participants were concerned about/were hesitant to use the drawing response 
interaction 
P2: “| wouldn’t use pictures on a test unless | had to. They take up too much time, 
and I’m bad at drawing.” 
P11: “I think if you had a time limit, [typing] it would be a lot faster.” 


P11: “Il want to do it on a piece of paper. | am frustrated by having to use the drawing 
response interaction.” 

P3: “If | couldn’t figure out the problem, | might draw, but otherwise, | would just use 
numbers.” 


11 of 12 participants struggled with the math content on one or more problems. This 
impacted their ability to assess the drawing response interaction 
e P6: “The (drawing) tools aren’t difficult, but the problem itself is confusing.” 
e@ 1of 12 participants (P4) was able to consistently and efficiently use the drawing 
response interaction to draw visual representations. She mentioned that she draws 
pictures in math at school. 


8 of 12 participants were unable to figure out the eraser tool 
e 5 of 6third grade participants were unable to figure out the “x” eraser tool 
e 3 of 6high school participants were unable to figure out the “x” eraser tool 
e Participants expected to be able to drag the eraser tool 
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Tasks 1 & 2: Using the drawing response interaction for Basic Tasks 
Participants were instructed to: 1) Use the pencil to draw a line in the box; 2) Use the pencil to 
draw a circle in the box. 


3rd Graders 


6 of 6 participants were able to use the drawing response interaction to draw a line and 
circle 


Rating 
e 5of 7 participants rated using the drawing response interaction for this task a 6 or 7 
(on ascale of 1 to 7, 1 being very difficult, 7 being very easy) 
e@ Average rating: 6.5 


Recommended features 
e Draggable Eraser Tool 
oO 3of 6 participants want a draggable eraser tool 
o “l| would like [the eraser] to work like this (makes swiping motion)--just 
swiping.” (P11) 
e Library of shapes to drag into the drawing response interaction (P2) 
e Tools to use in the drawing response interaction (ex: straightedge, compass) (P4) 


High Schoolers 


6 of 6 participants were able to use the drawing response interaction to draw a line and 
circle 


Rating 
e 6of 6 participants rated using the drawing response interaction for this task a 6 or 7 
e Average rating: 6.8 


Recommended features 
e Library of shapes to drag into the drawing response interaction (P1, P10) 
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Task 3: Using the Drawing Response Interaction for Basic Tasks 
Participants were instructed to: Draw a picture to show 1/2. 


3rd Graders 


6 of 6 participants were able to use the drawing response interaction to attempt this task 
e Not all participants drew a correct answer 


Rating 
e 30of5 participants rated using the drawing response interaction for this task a 6 or 7 
oO Note: 1 participant did not rate this question 
e Average rating: 5.6 
e P7 rated a2 because he didn’t know how to erase 


Recommended features 
e Library of shapes to drag into the drawing response interaction (P2) 
Oo P2: “For littler kids, it would be better if there were shapes you could add.” 


e Autocorrect shapes (P11) 


High Schoolers 


6 of 6 participants were able to use the drawing response interaction to attempt this task 
e Not all participants drew a correct answer 


Rating 
e 6of 6 participants rated using the drawing response interaction for this task a 6 or 7 
e Average rating: 6.8 


Recommended features 
@ Tool to automatically shade a shape (P9, P10, P12) 
e@ Tool to help draw a line in the middle of a shape (P12) 
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Third Grade Tasks: 


Task 4 
Participants were instructed to solve: There are 25 people going to a picnic in the park. There 
are 5 tables. One of the tables will be used for food. The other tables each can seat 6 people. 


3 of 6 participants did not understand the question, which may have impacted their ability 
to give feedback on the drawing response interaction (P2, P3, P7) 


Participants used the drawing response interaction/textbox in different ways 
e 3 of 6 participants only used the drawing response interaction (P7, P4, P3) 
e 2 of 6 participants used the drawing response interaction and textbox (P11, P8) 
e 1of6 participants only used the textbox (P1) 


One participant ran out of room in the drawing response interaction 
e P7: “Going outside the lines of the [drawing response interaction] was hard.” 


Rating 
e Solving: 3 of 6 participants rated solving this task a 6 or 7; 3 of 6 participants rated 
solving this task a 1 
Oo Average rating: 3.7 
e Drawing response interaction: 4 of 6 participants rated using the drawing response 
interaction for this task a 6 or 7 
© Average rating: 5.5 


Recommended Features 
e Library of shapes to drag into the drawing response interaction (P2, P7, P11, P8) 
e Ability to draw straight lines (P4) 
e Ability to stretch/shrink shapes (P2) 
e Ability to copy/paste shapes in the drawing response interaction (P2) 
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Task 5 

Participants were instructed to solve: Sophie is comparing the fractions 2/3 and 2/4. She says 
that 2/4 is bigger. Is she correct? Show your work or explain your answer using words, 
numbers, and pictures. 


2 of 6 participants ran out of room in the drawing response interaction (S8, $11) 
e P11: “Is there a way to make [my response] smaller?” 
e P8: “Oh, | think | need a bigger box!” 


5 of 6 participants used the drawing response interaction 
e 1of 6 participants used only the textbox (P11) 


Rating 

e Solving: 3 participants rated solving this task as a 6 or 7 
Oo Average rating: 5.2 

e Drawing response interaction: 2 of 6 participants rated using the drawing response 

interaction as a 6 or 7 
Oo Average rating: 5 
© participants rated lower than 6 due to issues with making their shapes and 
lines precise 


Recommended Features 
e@ Tool to automatically shade a shape (P4) 
o “Shading shapes is difficult.” (P4) 


Task 6 

Participants were instructed to solve: Robert is putting toy cars into 8 groups. One jar has 25 
toy cars and another jar has 23 toy cars. He wants the same number in each group. How many 
toy cars will be in each group? Show your work or explain your answer using words, numbers, 
and pictures. 


2 of 6 participants did not understand the question, which may have impacted their ability 
to give feedback on the drawing response interaction (P2, P7) 
e P2: “I’m confused because | know how many groups there are, but | don’t know what 
the groups are--are they the cars? 
e P7: “Il don’t get this one.” 


2 of 6 participants did not use the drawing response interaction for this problem 
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P3: “If | couldn’t figure out a problem, | might draw, but otherwise, | would use 
numbers.” 

P11: “It would take awhile for me to draw the cars. If | just write the answer, it will be 
a lot faster.” 


One participant ran out of room in the drawing response interaction 
e P4: “I would write my answer in the [drawing response interaction], but my box is 
full.” 


Rating 


e Solving: 3 of 6 participants rated solving this problem as a 6 or 7 
oO Average: 5 
e Drawing response interaction: 2 of 6 participants rated using the drawing response 
interaction as a 6 or 7 
oO Average: 5 
Oo P8 rated a3 because “you couldn’t really draw the dots in [the drawing 
response interaction].” 


Recommended Features 
e Library of shapes to drag into drawing response interaction (P2) 
e Ability to add labels to the drawing response interaction (P2) 


Task 7 


Participants were instructed to solve: A) Adam needs to put 19 pictures from Classroom A and 
23 pictures from Classroom B ona bulletin board. He wants to display the pictures in an array. 
Draw an array to represent the pictures on the bulletin board. There is space for up to 10 
pictures across and down on the bulletin board. B) Find the area of the array. 

4 of 6 participants drew something for Part A but were incorrect (S2, $7, $11, $3) 


participants had trouble drawing dots to represent pictures in the array 
e P11:“l wrote [the dots] kind of big, so | couldn’t fit them.” 


3 of 6 participants had to start over and erase due to lack of space or difficulties with 
drawing 
e@ P11: “Cause | accidentally did 7 (dots) instead of 10 and | wrote them kinda big so | 
couldn’t fit them.” 


Rating 


e Solving: 1 participant rated solving this problem a 6 or 7; 2 participants rated solving 
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this problem a 1 


© Note: One participant did not rate this question 


e Drawing response interaction: 0 participants rated the drawing response interaction a 
6 or 7; 3 participants rated the drawing response interaction a 3 
oO Note: One participant did not rate this question 
O P3: Rated a3 “because | had to erase and start over.” 
oO P7: Rated a1 “because | didn’t have enough lines and it didn’t make sense--it 
was hard to draw lines.” 


Recommended Features 
e Library of shapes to drag into drawing response interaction (P3) 
e Ability to draw straight lines (P4) 


Task 8 


Participants were instructed to solve: Tori and Leo set up their clubhouse with four tables. 
These rectangles represent the tabletops. A) Identify two tabletops with the same area, in 
square feet, and explain how you know the areas are equal. Enter your answers and your 

explanation in the space provided. 


3 of 6 participants sketched a picture/diagram (P2, P4, P7) 


2 of 6 participants were unable to answer the question 
e 1of6 participants began crying and stopped the test (P8) 
e@ 1of6 participants said “I’m not really sure how to do this” and was not able to sketch 
or type an answer (S11) 


Rating 
e Note: Only 2 participants provided ratings for this task. 


Recommended Features 
e Library of shapes to drag into drawing response interaction (P2, P3, P7) 
e Ability to draw straight lines (P2) 
e Ability to stretch/shrink shapes (P2) 
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Task 9 

Participants were instructed to solve: Martin cut a pan of cornbread into equal pieces as shown 
in the model. A) Martin gave % of the corn bread to his neighbor. Explain how you can use the 
model to show %. Then write a fraction that is equivalent to %. B) Martin gave 6/12 of the corn 
bread to his teacher. Write a comparison using <, >, or = to compare the fractions % and 6/12. 
Explain how the model can be used to compare these fractions. Enter your comparison and 
your explanation in the space provided. 


5 of 5 participants wanted the drawing response interaction to help them solve this 
problem 
e Note: For the duration of the tasks, there are only 5 participant participants. 


3 of 5 participants wanted to use the drawing response interaction to shade part of the 


given diagram (P7, P9, P11) 
3 of 5 participants wanted to draw their own representation of the pan (P2, P3, P4) 


Recommended Features 
@ Tool to automatically shade a shape (P2, P3, P7, P11) 


Task 10 

Participants were instructed to solve: Nolan has 16 pennies in one jar and 94 pennies in another 
jar. He uses some of the pennies to buy a pencil that costs 25 cents. What is the total number 
of pennies Nolan has left after he buys the pencil? Show your work. Enter your answer and 
your work in the space provided. 


4 of 5 participants wanted the drawing response interaction to help them solve this 
problem (P3, P4, P7, P11) 
e 1of5 participants prefered to use numbers, not pictures, to solve the problem (P2) 
o “It would be easier to solve the problem using just numbers and not the 


drawing response interaction” (P2) 


Recommended Features 
e Pictures of objects (like pennies) to drag into the drawing response interaction (P11) 
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Task 11 

Participants were instructed to solve: A) Shaun plotted a point on the number line by drawing 5 
equally spaced marks between 0 and 1 and placing a point on the third mark. He claims that 
the point represents the fraction % because each mark represents %, so the third mark 
represents %. Explain why Shaun’s reasoning is incorrect. Explain how you can use the number 
line to determine the fraction that Shaun’s point represents. Determine the fraction that 
Shaun’s point represents. B) Shaun wants to write a fraction that is equivalent to the fraction 
%s. Describe how Shaun can find a fraction that is equivalent to 7. 


3 of 5 participants wanted the drawing response interaction to help them solve this 
problem (P2, P4, P7) 
@ 1of5 participants indicated he would not use the drawing response interaction (P3) 
e 1of5 participants was not able to attempt the problem (P11) 


Participants wanted to use the drawing response interaction to represent the number line 
e Of the 3 participants who indicated they wanted to use the drawing response 
interaction: 
O 1of 3 participants wanted to draw on the given number line (P7) 
O 2 of 3 participants wanted to recreate the number line (P2, P4) 


Recommended Features 
e@ Tool to automatically add a number line (P2) 
e Ability to draw straight lines (P4) 
e Ability to add labels (P4) 


Task 12 

Participants were instructed to solve: Ms. Morales has a bag of beads. She gives Elena 5 beads. 
She gives Damian 8 beads more than Elena. She gives Trish 4 times as many beads as Damian. 
Ms. Morales then has 10 beads left in the bag. How many beads did Damian and Trish each 
receive? 


3 of 5 participants did not want to use the drawing response interaction to help them solve 
the problem (P2, P3, P7) 

e@ P2: “You could solve with just multiplication or addition.” 

e P7: “Il would just do it in my head and write the answer down.” 


Recommended Features 
e Library of shapes to drag into drawing response interaction (P4) 
e Ability to add labels (P4) 
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High School Tasks: 


Task 4 
Participants were instructed to solve: Draw a graph to show y=3x-1. 


4 of 6 participants were confused by the task and unable to complete the problem correctly 
e 2of 6 participants correctly completed the task (P6, P12) 


6 of 6 participants wanted a coordinate plane they could drag into the drawing response 
interaction 
e 5of 6 participants independently stated they wanted a coordinate plane to drag into 
the drawing response interaction 
e 1of6 participants, when prompted, stated he wanted a coordinate plane to drag into 
the drawing response interaction 


Rating 
e 5of6 participants rated using the drawing response interaction a 6 or 7 
e 1of6 participants rated using the drawing response interaction a 1 because he did 
not understand how to solve the problem (P9) 
e Average: 5.3 


Recommended Features 
e Graphs/grids to drag into the drawing response interaction (P1, P5, P6, P9, P12) 
e Ability to draw straight lines (P10, P12) 
e Ability to add “dots” to the drawing response interaction (P12) 


Task 5 
Participants were instructed to solve: Draw an isosceles triangle and the altitude. Mark equal 
sides and the right angle. 


6 of 6 participants attempted the task using the drawing response interaction 
e 2of 6 participants drew right isosceles triangles 
e 2of 6 participants used different colors to complete the task 


Rating 


e Solving: 5 of 6 participants rated solving the problem a 6 or 7 
oO Average: 5.8 
e Drawing response interaction: 4 of 6 participants rated using the drawing response 
interaction a 6 or 7 
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oO Average: 6.2 


Recommended Features 
e@ Tools to use in drawing response interaction (ex: straightedge, compass) (P1, P5) 
e Ability to draw straight lines (P5, P10, P12) 
e Library of shapes to drag into drawing response interaction (P12) 


Task 6 

Participants were instructed to solve: In a system of two linear equations, the lines represented 
by each equation have the same slope. A) What could be the total number of solutions to the 
system of equations? B) Explain why you chose your answer in Part A. 


1 of 6 participants completed the problem completely and correctly (P10) 


5 of 6 participants used the drawing response interaction and the textbox to answer the 
question (P1, P6, P9, P10, P12) 
e 1of 6 participants used only the drawing response interaction (P5) 


Rating 
e Solving: 3 of 5 participants rated solving this problem a 6 or 7 
© Note: One participant did not rate this question 


Oo Average: 5.4 
e Drawing response interaction: 5 of 6 participants rated using the drawing response 
interaction a 6 or 7 
Oo Note: One participant did not rate this question 
oO Average: 6.6 


Recommended Features 
e@ Graphs/grids to drag into the drawing response interaction (P6) 
e Ability to add labels (P12) 
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Task 7 
Participants were instructed to solve: Kim made a claim about right triangles. If the hypotenuse 
of one right triangle is congruent to the hypotenuse of another right triangle, then the two 


triangles are congruent. State whether Kim’s claim is correct or incorrect. Provide either an 
explanation or a counterexample to support your position. 


6 of 6 participants attempted the problem 


5 of 6 participants used the drawing response interaction and the textbox to answer the 
question (P1, P5, P6, P9, P10) 
e 1of6 participants only used the textbox (P12) 


Rating 
e Solving: 4 of 6 participants rated solving the problem as a 6 or 7 
oO Average: 5.8 
e Drawing response interaction: 5 of 6 participants rated using the drawing response 
interaction as a 6 or 7 
Oo Average: 6 


Recommended Features 
e Library of shapes to drag into drawing response interaction (P1) 
e@ Tools to use in drawing response interaction (ex: straightedge, compass) (P1) 
e Ability to add labels (P1, P12) 
e Ability to copy/paste shapes in the drawing response interaction (P10) 


Task 8 
Participants were instructed to solve: Marcella drew each step of a construction of an angle 
bisector. Use the steps in the construction to prove that... 


5 of 6 participants used the drawing response interaction to re-sketch part 5 of Marcella’s 
construction (P1, P6, P9, P10, P12) 
e 1of6 participants used both the drawing response interaction and the textbox (P1) 
e 1of 6 participants was not sure how to show his work for the problem (P5) 


Rating 


e Solving: 1 of 5 participants rated solving the problem a 6 or 7; 4 of 5 participants rated 
solving the problem a 2 or 3 
© Note: One participant did not rate this question 
Oo Average: 3.4 
e Drawing response interaction: 2 of 5 participants rated using the drawing response 
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interaction a 6 or 7; 3 of 5 participants rated using the drawing response interaction a 
5 


Oo Average: 5.6 


Recommended Features 
e@ Tools to use in drawing response interaction (ex: straightedge, compass) (P1) 
e Ability to add labels (P6) 
e Ability to draw straight lines (P6) 


Task 9 

Participants were instructed to solve: The figure shows line RS parallel to line UV. The lines are 
intersected by 2 transversals. All lines are in the same plane. A) Explain why triangle RTS is 
similar to triangle VTU. B) Given that m<STV=108 degrees, determine m<SRT+m<TUV. 


4 of 6 participants wanted to use the drawing response interaction to help them solve the 
problem (P1, P5, P6, P10) 
P10: “...1 would like a drawing response interaction so | can make up angle 
measurements to make sure it would all work out.” 
P2: “Even if there was an option to draw it out, | would much prefer to write it out.” 
P9: “I don’t think you would need a drawing response interaction for Part A, because 
you are just explaining, and the picture is already right there.” 


4 of 6 participants wanted the ability to label the given shape (P1, P5, P6, P10) 


Task 10 

Participants were instructed to solve: Let the absolute value of x plus the absolute value of y 
equal c, where cis a real number. Determine the number of points that would be on the graph 
of the equation for each given case: Case 1: c<0; Case 2: c=0, Case 3: c>0. 


3 of 6 participants wanted to use the drawing response interaction to help them solve the 
problem (P1, P10, P12) 
e P10: “I would like a drawing response interaction just so | could write out my 
calculations.” 
e P5: “Il would write my answer out, | wouldn’t use the drawing response interaction.” 


At this point in the test, participants were having trouble comprehending the content of the 
questions 
e@ P9: “If | knew the actual math, it would make more sense.” 
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Recommended Features 
e Graphs/grids to drag into the drawing response interaction (P1, P12) 


e P12: “If there were an included graph, | would use the drawing response interaction. 
If not, | wouldn’t.” 


Task 11 
Participants were instructed to solve: Do the points on the line y=3x-2 have a constant ratio of 
the y-coordinate to the x-coordinate for any point on the line except the y-intercept? 


5 of 6 participants wanted to use the drawing response interaction to help them solve the 
problem (P1, P5, P6, P9, P10) 
P6: “| would want a drawing response interaction with a graph so | could prove what it 
is before | explain my answer.” 
P9: “I think this one would be really good if you had the drawing box and could add a 
graph...math is easier using pictures so you can see what you are doing.” 
P10: “I would like to be able to graph this, that would help me.” 


Recommended Features 
e@ Graphs/grids to drag into the drawing response interaction (P1, P6, P9, P10) 


Task 12 

Participants were instructed to solve: Consider the following claim: If the point (2+d,y) is on the 
graph of the function f(x)=x(x-4), then the point (2-d, y) is also on the graph. Use algebra to 
show that the claim is true. What is the relationship between the line x=2 and the graph f(x)? 


5 of 6 participants wanted to use the drawing response interaction to help them solve the 
problem (P1, P5, P6, P9, P10) 
e P6: “I would write down the equations and solve it.” 
e@ P12: “I probably wouldn’t use a drawing response interaction here. | might use the 
drawing response interaction if it were more accurate and precise.” 


Recommended Features 
e Graphs/grids to drag into the drawing response interaction (P1, P9) 
e Ability to handwrite work/steps (P6, P10) 
o “I definitely need to be able to have something to write out my work here.” 
(S10) 
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Task 13 

Participants were instructed to solve: A dilation centered at point C with a scale factor of k, 
where k>0, can be defined as follows: 1) The image of point C is itself. The is, C’=C. 2) For any 
point P other than C, the point P’ is on CP, and CP’=k*CP. Use this definition and the diagram 
shown to prove the following theorem: If A’B’ is the image of AB after a dilation centered at 
point C with a scale factor of k, where k>0, then A’B’=k*AB. 


3 of 6 participants wanted to use the drawing response interaction to help them solve the 
problem (P6, P9, P10) 
e P9: “You don’t need a drawing area for this. You just have to explain your answer in 
the typing box.” 
e P10: “I’d like to have a thing on the side so | could write out my work and then 


another one for a final answer so | can draw that out to go along with my 
explanation.” 


Recommended Features 
e Ability to label/draw on the given diagram (P6) 
e Ability to handwrite steps (P10) 


Task 14 

Participants were instructed to solve: A city plans to implement a composting program. In the 
composting program, food waste will be collected from the residents and sent to one of these 
compost collection sites [Table]. A) Based on the given information, determine which 
composting collection site is cheapest. 


4 of 6 participants wanted to use the drawing response interaction to help them solve the 
problem (P1, P6, P9, P10) 
P1: “I want a (drawing) tool to help me map out of the information.” 
P6: “Il want a drawing response interaction to help me work out my multiplication and 
division--a blank space to handwrite my work.” 
P9: “You might need an area to draw your model. Maybe an area to redo the [table].” 
P10: “I would definitely need a workspace so | could write it all out. | don’t think | 
could do this one without drawing it.” 


One participant wanted a larger drawing response interaction 
e P41: “I want a bigger or expandable [drawing response interaction].” 


Recommended Features 
e Ability to handwrite steps (P6, P10) 
e Blank table to write in (P6, P9) 
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e Ability to expand the drawing response interaction (P1) 


o P41: “| want a bigger or expandable [drawing response interaction].” 


Physical Interaction with the Computer & iPad 


Overall, participants were able to navigate both the computer and the iPad, with a few 
minor issues 


3 of 6 participants who used the iPad for testing indicated they would rather use a 
computer/mouse than an iPad 
P11: “I’d rather do this on a computer with a mouse (rather than on an iPad).” This 
participant became frustrated during the test and gave up on multiple tasks. 
P11: “I like computers better (than iPads) because the iPad has a lag.” 
P6: “It’s difficult for some people to use the mouse to write, but | have experience 
with it and don’t mind.” 
P10: “I don’t mind using the iPad, but it seems like everything | am at school, there’s 
always something wrong [with the iPads]. I’d be worried about it.” 


1 of 6 participants struggled with using the mouse 
e P3: “Using the mouse is hard. It’s wiggly!” 


A few participants struggled with some minor iPad issues 
e 2of 6 participants accidentally opened the keyboard and were not sure how to close 
it. 
e 1of6 participants accidentally highlighted parts of the question when he was trying to 
draw. He was unsure how to select the highlighted parts. 
e Multiple participants accidentally zoomed when they were trying to erase or draw. 


Participants did not specifically comment about line weight or smoothness 
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FINDINGS & RECOMMENDATIONS 


This section discusses positive results and potential issues that became apparent during the 
study and provides actionable design recommendations to increase the usability of the drawing 
response interaction. 


Overall, participants were comfortable interacting with the drawing response interaction for 
simple tasks. Some participants had trouble using the drawing response interaction with some 
of the more complex math problems. 


Objective 1: Can participants use the drawing response interaction to 
help them solve math problems? 


General use of drawing response interaction 
e@ 12 of 12 participants were able to complete simple tasks (drawing a line and a circle) 
using the drawing response interaction 


Third grade participants experienced issues with the size of the drawing response 
interaction 
e 5 of 6 third grade participants had difficulty keeping their drawings within the drawing 
response interaction 
oO P4: “| would write my answer in the drawing box, but my box is full.” 
Oo P41: “Il want a big enough drawing tool for all of my information to fit in.” 
e Third grade participants ran out of room on 5 different tasks 


oO 1of 6 third grade participants ran out of room on task 4 (P7) 
2 of 6 third grade participants ran out of room on task 5 (P8, P11) 
1 of 6 third grade participants ran out of room on task 6 (P4) 
3 of 6 third grade participants ran out of room on task 7 (P3, P7, P11) 
1 of 6 third grade participants ran out of room on task 8 (P3) 


Eraser Functionality 
e 5of 12 participants verbally pointed out the “x” eraser tool, but 8 of 12 participants 
were unable to figure out how to use the eraser tool 
e Participants expected to be able to erase by dragging the mouse/their finger 
e@ P11: “Il would want a trashcan if | mess up everything. A square that wipes away like 
the eraser tool to erase specific things.” 
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Concerns with inaccuracy 
e@ 7 of 12 participants commented on and were concerned by the inaccuracy of their 
shapes/drawings 
oO S10: "[The drawing response interaction] is lagging a bit and | can't get my lines 
right." 


Confusion about Use of Drawing Response Interaction versus Textbox 
e 60f 12 participants indicated being unsure of where to put their answer and/or 
whether to use both the drawing response interaction and the textbox 

oO 1of 12 participants was unsure of where to put a final answer (P4) 

Oo 1of 12 participants wants a separate tool for work and answer (P10) 

oO P12: “I’m guessing since the boxes (drawing response interaction and textbox) 
are next to each other, | can use one or the other. If they were on top of each 
other, I’d need to use both.” 

3 of 12 participants indicated that they wanted the textbox and palette to be better 
emphasized 

1 of 12 participants suggested having just one box in which you could both draw and 
type (P2) 


Rigor of Questions 


e@ The third graders and high schoolers struggled with the rigor of the math questions, 
which may have impacted their ability to assess and provide recommendations for the 
drawing response interaction 


Third grade participants experienced issues with the size of the drawing response 
interaction 
oO Create the ability to expand the drawing response interaction 
o Adjust the size of the drawing response interaction based on the demands of 
the question; provide more space for problems requiring more handwritten 
work 
Eraser Functionality 
oO Include two types of erasing capability: a) a tool which erases the entire 
drawing response interaction box (ex: a trashcan icon), b) a draggable eraser 
tool which enables users to flexibly erase portions of their work 
Concerns with inaccuracy 
oO Include a library of shapes which can be dragged into the drawing response 
interaction 
m Allow users to create accurate shapes 
m Eliminate the user worry that being a poor artist will impact their 
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scores 
e Confusion about Use of Drawing Response Interaction versus Textbox 
oO Include a one- or two-question interactive tutorial at the beginning of the test 
that enables participants to practice using the drawing response interaction 
and textbox/palette in a low-stakes manner 
oO. Include clear directions on how participants are supposed to interact with the 
drawing response interaction and/or textbox 
oO Clearly state whether participants are required to use the drawing response 
interaction, the textbox, or both 
oO Include a clearly designated space for participant answers 
e Rigor of Questions 
o For future tests, update the included math problems to lower the intrinsic 
cognitive load of the tasks 
m Forsome participants, the intrinsic cognitive load of the problems was 
high enough that they were not able to consider how they would use 
the drawing response interaction 
Include less rigorous (yet still rich and complex) problems to give participants a 
chance to better analyze and assess the drawing response interaction 
Requiring the participants to show their solution in the drawing response 
interaction seems to increase the complexity of the task. More than half of the 
third graders struggled when trying to draw pictures of increased complexity, 
e.g. the problem where they were expected to represent 5 tables and 6 people 
sitting around them, and the 10x10 array 


Objective 2: What features will participants want to add to the 
drawing response interaction? 


Below is an overview of the features recommended by the participants, in ranked order from 
most recommended to least recommended 


e 1.) Graphs/grids to drag into the drawing response interaction 


Oo Recommended by 7 of 12 participants in 18 different instances 
o Recommended by by 2 of 6 third graders 
o Recommended by 5 of 6 high schoolers 
e 2.) Library of shapes to drag into drawing response interaction 
o Recommended by 8 of 12 participants in 20 different instances 
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o Recommended by 4 of 6 third graders 
Oo Recommended by 4 of 6 high schoolers 
Oo Note: 4 participants specifically wanted to be able to add “dots” 
3.) Ability to label/markup a provided shape/graph (ex: a shape included in the task 
prompt) 
o Recommended by 8 of 12 participants in 11 different instances 
o Recommended by 4 of 6 third graders 
o Recommended by 4 of 6 high schoolers 
4.) Ability to draw straight lines 
Oo Recommended by 7 of 12 participants in 12 different instances 
o Recommended by 3 of 6 third graders 
o Recommended by 4 of 6 high schoolers 
5.) Tool to automatically shade a shape 
o Recommended by 7 of 12 participants in 7 different instances 
o Recommended by 4 of 6 third graders 
o Recommended by 3 of 6 high schoolers 
6.) Ability to add labels to the drawing response interaction 
Oo Recommended by 5 of 12 participants in 8 different instances 
o Recommended by 2 of 6 third graders 
o Recommended by 3 of 6 high schoolers 
7.) Ability to handwrite steps/work 
Oo Recommended by 5 of 12 participants in 7 different instances 
o Recommended by 2 of 6 third graders 
o Recommended by 3 of 6 high schoolers 
8.) Tools to use in drawing response interaction (ex: straightedge, compass) 
o Recommended by 4 of 12 participants in 4 different instances 
Oo Recommended by 2 of 6 third graders 
o Recommended by 2 of 6 high schoolers 
9.) Ability to copy/paste 
o Recommended by 3 of 12 participants in 3 different instances 
o Recommended by 1 of 6 third graders 
o Recommended by 2 of 6 high schoolers 
10.) Ability to stretch/shrink shapes 
o Recommended by 2 of 12 participants in 3 different instances 
o Recommended by 2 of 6 third graders 
o Recommended by 0 of 6 high schoolers 
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Recommendations 


Shapes 
e Adda library of shapes which allows participants to add a variety of shapes (square, 
rectangles, triangles, lines, “dots,” etc.) into the drawing response interaction 
oO Enable participants to drag shapes into the drawing response interaction 
oO Enable participants to stretch/shrink shapes by dragging 


Graphs 
e Adda library of graph types which allows participants to add a variety of graphs (grids, 
coordinate planes of different sizes, number lines, etc.) into the drawing response 
interaction 
oO Enable participants to drag graphs into the drawing response interaction 
oO For questions requiring graphing, include the coordinate plane/grid in the 
drawing response interaction 
Shading 
e@ Add a shading tool which allows participants to shade a shape or part of a shape 
© participants recommended a spray can icon 
Labeling 
e Allow participants to label/markup graphs/images included in a problem or allow 
participants to copy/paste a given graph/image into the drawing response interaction 
© participants expected to be able to draw directly on given images 
Add a labeling tool which allows participants to drag letter and number labels into the 
drawing response interaction 
Ability to Handwrite Steps/Work 
e Increase drawing response interaction box size to enable participants to more easily 
show their work 
0. Third graders in particular struggled to keep their work within the box 
Tools 
e Add aset of tools to help participants more accurately show their work 
®@ participants expected to see a straightedge, protractor, and compass 
Copy/Paste 
e Allow participants to copy/paste shapes from their drawing response interaction box 
® participants want to, for example, draw a line, then copy and paste it so they can 
ensure they have two lines of the same length. 


PARCC Drawing Response Interaction Usability Test 
31 


APPENDIX 1 — MODERATOR’S GUIDE 


1. Introductions — introduce team members; Rusty, Kimberly, Susan, Lulu 
2. Room Setup — we'll be recording the session. 


3. Purpose — today you’re going to be looking at a new concept to help participants 
become better prepared for college and career. 


4. Feedback — we'll be going through a series of activities. We want you to think 
out loud and feel comfortable sharing what you're thinking. If you are feeling 
confused, or you have expectations that are not being met, we want to hear about 
that. Be as honest as possible. If you think something is awful, please say so. Don't 
be shy: you won’t hurt anyone’s feelings. Please remember that you may disagree 
with someone else in the room, and that’s okay. Just keep the conversation honest 
and respectful. 


5. Our goal is to make well-designed products for people like you, so it’s important 
we understand what you think and what does and doesn’t work for you. 


6. Any Questions? 


Overview 


Demographic Interview: Do you, or have you ever use a computer to complete math 
assignments? 


Scenario: Imagine that you’re about to take a test online. You will be expected to show 
all of your work within the test on the computer. 


Tasks 


1. drawing response interaction (G3 Q 1-3; G9-12 Q 1-5) 
o Can users successfully use the drawing response interaction to complete the 
task? 
o Dousers identify any other functionality they would like to see added to the 
drawing response interaction? 


a. [After login] Look over this page and tell me what you think you can do here. 


PARCC Drawing Response Interaction Usability Test 
32 


b. Go ahead and do what the question asks. 


[SEQ Survey: [Insert task] Overall this task was? (rating 1-7)] 


2. Multi-step Questions (G3 Q 4-8; G9-12 Q 6-8) 
o Can users successfully use the drawing response interaction to complete the 


task? 
o Do.users identify any other functionality they would like to see added to the 


drawing response interaction? 


c. Here’s an actual math problem you might encounter in a test. How would you 
solve it using the tools provided? 


1) How would you use these tools to solve the problem? 
2) What are you thinking? 


[If participants get bogged down with equation editor, redirect them to just use 
drawing response interaction] 


6) Is there anything missing, or would you like something to act/behave 


differently? 
7) What would you add to the drawing response interaction in order to help 


you solve this problem? 
8) What do you think of the location of the tool? 


[Depending on time, continue through remaining questions, or skip to next 
section] 


[SEQ Survey aligned to content: [Insert task] Overall this task was: (rating 1-7)] 


PARCC Drawing Response Interaction Usability Test 
33 


[SEQ Survey aligned to drawing response interaction: [Insert task] Overall this task was: 
(rating 1-7)] 


3. Questions drawing response interaction (G3 Q 9-12; G9-12 Q 9-15) 
o Dousers identify any other functionality they would like to see added to the 
drawing response interaction? 


d. So, these math problems don’t have a drawing response interaction. You don’t 
have to actually solve the problem, but can you tell us how the drawing response 
interaction might help you to answer these questions? 


1) How would the drawing response interaction help you solve the problem 
or help explain your answer? 

2) What would you add to the drawing response interaction in order to help 
you solve this problem? 


[If participants get bogged down with equation editor, redirect them to just focus 
on features of the drawing response interaction] 


3) Is there anything missing, or would you like something to act/behave 
differently? 


4. Post-interview 


[If not addressed in earlier conversations about drawing response interaction] 


e. Are there any online math tools, apps, etc. that you’ve used before that help you 
explain your work, or solve math problems? 


1) If so, what are they? How do they help you? 

2) What can we do to make it easier for you to show your work/solve the 
problems? 

3) [If not already addressed earlier] What would you add to the drawing 
response interaction in order to help you solve math problems? 
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APPENDIX 2--PARTICIPANT VIDEOS 


Each participant has a video outlining their interaction with the PARCC drawing response 
interaction. 


All videos can be accessed in the folder PARCC Usability Sessions July 105- Videos, found at 
https://pearson.box.com/s/6kt4pfs750zumwmr3dbeuzqgizyu2k5ib. 


